Camera localization, i.e., recovering the three-dimensional (3D) position and orientation of a moving camera is one of the fundamental tasks in computer vision with a wide variety of applications in robot navigation, car localization for autonomous driving, device localization for mobile navigation, and augmented/virtual reality. A key component in camera localization is the concept of a map. A map is an abstract summary of the input data that establishes geometric constraints between observations and can be used to establish correspondences between consecutive image frames. A map can be queried to obtain the camera pose for correcting drift in relative pose estimation and reinitialize the camera pose if the tracking is lost. Maps, however, are usually defined in an application-specific manner with hand-crafted features. Examples include 3D landmarks, 3D points, line/edge structures for indoor/man-made scenes, groups of pixels with depth, or object-level context for semantic techniques. Being application-specific, the map representations may ignore useful (sometimes, the only available) features in environments and thus may not be optimal or robust for general scenarios. There is a need for addressing these issues and/or other issues associated with the prior art.